Self-Driving Car Engineer Nanodegree

Deep Learning

Project: Build a Traffic Sign Recognition Classifier

In this notebook, a template is provided for you to implement your functionality in stages which is required to successfully complete this project. If additional code is required that cannot be included in the notebook, be sure that the Python code is successfully imported and included in your submission, if necessary. Sections that begin with 'Implementation' in the header indicate where you should begin your implementation for your project. Note that some sections of implementation are optional, and will be marked with 'Optional' in the header.

In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. In addition, Markdown cells can be edited by typically double-clicking the cell to enter edit mode.


Step 1: Dataset Exploration

Visualize the German Traffic Signs Dataset. This is open ended, some suggestions include: plotting traffic signs images, plotting the count of each sign, etc. Be creative!

The pickled data is a dictionary with 4 key/value pairs:

  • features -> the images pixel values, (width, height, channels)
  • labels -> the label of the traffic sign
  • sizes -> the original width and height of the image, (width, height)
  • coords -> coordinates of a bounding box around the sign in the image, (x1, y1, x2, y2). Based the original image (not the resized version).
In [1]:
import numpy as np
import tensorflow as tf
import cv2
import matplotlib.pyplot as plt
import time
import csv
import itertools
import os
from scipy import ndimage
from tqdm import tnrange
%matplotlib inline
In [2]:
# Load pickled data
import pickle

# TODO: fill this in based on where you saved the training and testing data
training_file = '../../sdc/traffic-signs-data/train.p'
testing_file = '../../sdc/traffic-signs-data/test.p'

with open(training_file, mode='rb') as f:
    train = pickle.load(f)
with open(testing_file, mode='rb') as f:
    test = pickle.load(f)
    
X_train, y_train = train['features'], train['labels']
X_test, y_test = test['features'], test['labels']

# Save Original Test for future evaluation
X_test_orig = X_test
y_test_orig = y_test
In [3]:
# Load Sign Names

def read_signnames(file):
    with open(file) as f:
        csvreader = csv.reader(f)
        header = next(csvreader)
        signnames = {}
        for row in csvreader:
            signnames[int(row[0])] = row[1]
        return signnames

signnames = read_signnames('signnames.csv')
print("Sign names loaded: len = ", len(signnames))
Sign names loaded: len =  43
In [4]:
print('X_train.shape =', X_train.shape)
print('y_train.shape =', y_train.shape)
print('X_test.shape =', X_test.shape)
print('y_test.shape =', y_test.shape)
print('image shape = ', X_train[0].shape)
np.max(y_train)+1
X_train.shape = (39209, 32, 32, 3)
y_train.shape = (39209,)
X_test.shape = (12630, 32, 32, 3)
y_test.shape = (12630,)
image shape =  (32, 32, 3)
Out[4]:
43
In [5]:
### To start off let's do a basic data summary.

# TODO: number of training examples
n_train = X_train.shape[0]

# TODO: number of testing examples
n_test = X_test.shape[0]

# TODO: what's the shape of an image?
image_shape = X_train[0].shape

# TODO: how many classes are in the dataset
n_classes = np.max(y_train) + 1

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)
Number of training examples = 39209
Number of testing examples = 12630
Image data shape = (32, 32, 3)
Number of classes = 43
In [6]:
### Data exploration visualization goes here.
### Feel free to use as many code cells as needed.


# Show all classes
def show_all_classes(data, labels, num_line=9, num_imgs=3):
    for c in range(n_classes):
        if c * num_imgs % num_line == 0:
            plt.figure((c * num_imgs) // num_line, figsize=(18, 2))
        s = data[labels == c]
        indices = np.random.randint(0, len(s), num_imgs)

        for n, idx in enumerate(indices):
            plot_num = (c * num_imgs) % num_line + n + 1
            plt.subplot(1, num_line, plot_num)
            cur_axes = plt.gca()
            cur_axes.axes.get_xaxis().set_ticks([])
            cur_axes.axes.get_yaxis().set_ticks([])
            plt.imshow(s[idx])
            signname = signnames[c][:17] + '..' if len(signnames[c]) > 17 else signnames[c]
            plt.title('%d - %s' % (c, signname), fontsize=10)
            
show_all_classes(X_train, y_train)
In [7]:
# Draw the classes distribution in X_train samples

def show_classes_distribution(labels, title = None):
    # Count classes samples
    uniq_labels = sorted(set(labels.tolist()))
    n_labels = len(uniq_labels)
    class_counts = np.zeros([n_labels])
    for c in uniq_labels:
        class_counts[c] = np.sum(labels == c)
    y_pos = np.arange(n_labels)
    plt.figure(figsize=(18, 5))
    plt.bar(uniq_labels, class_counts)
    plt.xticks(y_pos)
    plt.ylabel('Classes Counts')
    if title: plt.title(title)
    
show_classes_distribution(y_train, title='Train Classes Distribution')
show_classes_distribution(y_test, title='Test Classes Distribution')

Step 2: Design and Test a Model Architecture

Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.

There are various aspects to consider when thinking about this problem:

  • Your model can be derived from a deep feedforward net or a deep convolutional network.
  • Play around preprocessing techniques (normalization, rgb to grayscale, etc)
  • Number of examples per label (some have more than others).
  • Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [8]:
### Preprocess the data here.
### Feel free to use as many code cells as needed.

def grayscale(img):
    """Applies the Grayscale transform
    This will return an image with only one color channel
    but NOTE: to see the returned image as grayscale
    you should call plt.imshow(gray, cmap='gray')"""
    return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)


def data_to_grayscale(data):
    # Transform to grayscale
    shape = data.shape
    data_gray = np.zeros((shape[0], shape[1], shape[2]))
    for idx, image in enumerate(data):
        data_gray[idx] = grayscale(image)
    return data_gray


def random_rotate(img):
    angle = np.random.randint(-25, 25)
#     print('angle =', angle)
    return ndimage.rotate(img, angle, reshape=False)

def random_gaussian_filter(img):
    sigma = np.random.choice([0.2,0.5,0.7,1])
#     sigma = 0.5
#     print('sigma =', sigma)
    return ndimage.gaussian_filter(img, sigma=sigma)

def random_noise(img):
    noise = np.zeros_like(img)
    noise_lvl = np.random.choice([10, 20, 30])
    cv2.randn(noise, 0, noise_lvl.flatten())
#     print("noise =", noise)
    return img + noise

def random_image_transform(img):
    transforms = {
        0: random_rotate,
        1: random_gaussian_filter,
        2: random_noise
    }
    n = np.random.randint(3)
    im = transforms[n](img)
#     im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)
    return im

    

"""
X_train_gray = data_to_grayscale(X_train)

# Show random images from X_train_gray
n_samples = 12
n_samples_idx = np.random.randint(0, len(X_train_gray), n_samples)
gray_samples = X_train_gray[n_samples_idx]
plt.figure(figsize=(18, 5))
for n, img in enumerate(gray_samples):
    plt.subplot(1, n_samples, n + 1)
    cur_axes = plt.gca()
    cur_axes.axes.get_xaxis().set_ticks([])
    cur_axes.axes.get_yaxis().set_ticks([])
    plt.imshow(img, cmap='gray')
"""
Out[8]:
"\nX_train_gray = data_to_grayscale(X_train)\n\n# Show random images from X_train_gray\nn_samples = 12\nn_samples_idx = np.random.randint(0, len(X_train_gray), n_samples)\ngray_samples = X_train_gray[n_samples_idx]\nplt.figure(figsize=(18, 5))\nfor n, img in enumerate(gray_samples):\n    plt.subplot(1, n_samples, n + 1)\n    cur_axes = plt.gca()\n    cur_axes.axes.get_xaxis().set_ticks([])\n    cur_axes.axes.get_yaxis().set_ticks([])\n    plt.imshow(img, cmap='gray')\n"

Question 1

Describe the techniques used to preprocess the data.

Answer:

In [9]:
# Combine train and test into one dataset
# X = X_train
# y = y_train

X = np.concatenate((X_train, X_test), axis=0)
y = np.concatenate((y_train, y_test), axis=0)
In [10]:
### Generate data additional (if you want to!)
### and split the data into training/validation/testing sets here.
### Feel free to use as many code cells as needed.

# Generate additional data hints
# real-world variabilities such as viewpoint variations, lighting conditions
# (saturations, low-contrast), motion-blur, occlusions, sun
# glare, physical damage, colors fading, graffiti, stickers and
# an input resolution as low as 15x15






def generate_additional_images(X, y):
    # Count classes samples
    uniq_labels = sorted(set(y.tolist()))
    n_labels = len(uniq_labels)
    class_counts = np.zeros([n_labels])
    
    all_ind = np.arange(len(y))
    img_indices = {}
    for c in uniq_labels:
        class_counts[c] = np.sum(y == c)
        img_indices[c] = all_ind[y == c]
    
    print("len_X =", len(X))
    print("len_y =", len(y))
    print("class_counts =", class_counts)
        
    max_num = np.max(class_counts)
    min_num = np.min(class_counts)
    print("max_num =", max_num)
    print("min_num =", min_num)

    errors = 0
    
    classes_range = tnrange(n_labels, desc="Classes") # tqdm progress
    
    for c in classes_range:
        c_num = np.sum(y == c)
        num = max_num - c_num
#         print('%2d: c_num = %d, num = %d' % (
#             c, c_num, num))
        nimgs = []
        nimgs_labels = []
        num_range = tnrange(int(num), desc="C=%d" % c) # tqdm progress
        
        for n in num_range:
            # Draw random image of class argmin
            sample = np.random.choice(img_indices[c])
#             print("sample =", sample)
            # Transform it
            try:
                nimg = random_image_transform(X[sample])
            except:
                errors += 1
                nimg = random_gaussian_filter(X[sample])
#                 continue
#                 print('Error! c = %d, sample = %d' % (c, sample))
            # Append to X and append to y
            nimgs.append(nimg)
            nimgs_labels.append(c)
            
        if len(nimgs) > 0:
#             print("1: len_X =", len(X))
#             print("1: X.shape =", X.shape)
#             print("1: len_nimgs =", len(nimgs))
    #         print("1: nimgs.shape =", nimgs[0].shape)
            X = np.append(X, np.array(nimgs), axis=0)
#             print("1: len_X =", len(X))
            plt.imshow(nimgs[0])
            y = np.append(y, nimgs_labels)
            class_counts[c] += num

            
#             
#         print("all_ind =", img_indices[c])
#         print("sample =", np.random.choice(img_indices[c]))
#         max_num - min_num > 0.1 * max_num
        # add sample of class argmin
#         argmin = np.argmin(class_counts)
#         print("argmin =", argmin)
        
        
        # Append to X and append to y
        # +1 to class_counts
#         max_num = np.max(class_counts)
#         min_num = np.min(class_counts)
#         print("max_num =", max_num)
#         print("min_num =", min_num)
#         break
        
#     print("len_X =", len(X))
#     print("len_y =", len(y))
#     print("class_counts =", class_counts)
    print("errors =", errors)
    return X, y
    
X1, y1 = generate_additional_images(X, y)
show_classes_distribution(y1, 'All Dataset')

# Visualize sample of each class
show_all_classes(X1, y1, num_line=9, num_imgs=3)


# Split Data into Train/Validation/Test
def train_validation_test(X, y):
    # Shuffle it
    r_idx = np.random.permutation(len(X))
    print("len_X =", len(X))
    print("len_y =", len(y))
#     print("r_idx =", r_idx)
    X = X[r_idx]
    y = y[r_idx]

    # Train/Valid/Test distributions
    p_train = 0.6
    p_valid = 0.2
    p_test = 0.2

    # Sanity check
    assert p_train + p_valid + p_test == 1.0
#     assert p_train + p_valid == 1.0

    N = len(X)
    train_idx = int(N*p_train)
    valid_idx = int(N*(p_train+p_valid))
    X_train = X[:train_idx]
    y_train = y[:train_idx]
    X_valid = X[train_idx:valid_idx]
    y_valid = y[train_idx:valid_idx]
    X_test = X[valid_idx:]
    y_test = y[valid_idx:]

#     X_test = X[valid_idx:]
#     y_test = y[valid_idx:]
                
    assert len(X_train) + len(X_valid) + len(X_test) == N
    assert len(y_train) + len(y_valid) + len(y_test) == N
#     assert len(X_train) + len(X_valid) == N
#     assert len(y_train) + len(y_valid) == N

    
    return X_train, y_train, X_valid, y_valid, X_test, y_test
#     return X_train, y_train, X_valid, y_valid

X_train, y_train, X_valid, y_valid, X_test, y_test = train_validation_test(X1, y1)
# X_train, y_train, X_valid, y_valid = train_validation(X1, y1)

show_classes_distribution(y_train, 'Train + Generated')
show_classes_distribution(y_valid, 'Validation + Generated')
show_classes_distribution(y_test, 'Test + Generated')
len_X = 51839
len_y = 51839
class_counts = [  270.  2940.  3000.  1860.  2640.  2490.   570.  1890.  1860.  1950.
  2670.  1740.  2790.  2880.  1050.   840.   570.  1470.  1590.   270.
   450.   420.   510.   660.   360.  1980.   780.   300.   690.   360.
   600.  1050.   300.   899.   540.  1590.   510.   270.  2760.   390.
   450.   300.   330.]
max_num = 3000.0
min_num = 270.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
errors = 0
len_X = 129000
len_y = 129000
In [11]:
# plt.imshow(random_rotate(X[3756]))
In [12]:
# Normalize data
def normalize(data):
    pixel_depth = 255
    return (data - pixel_depth / 2) / pixel_depth

X_train_norm = normalize(X_train)
X_valid_norm = normalize(X_valid)
X_test_norm = normalize(X_test)

print("Data normalized.")

# print("X_train[0] =", X_valid[0])
Data normalized.

Question 2

Describe how you set up the training, validation and testing data for your model. If you generated additional data, why?

Answer:

In [13]:
### Define your architecture here.
### Feel free to use as many code cells as needed.
In [14]:
# Config

class SmallConfig(object):
    """Small config. """
    # init_scale = 0.1
    image_size = 32
    num_classes = 43
    num_channels = 3
    fc1_size = 512
    batch_size = 128 # 128
    max_epoch = 35 # 250 45 55 - latest
    keep_prob = 0.5
    lr = 0.01
    lr_decay = 0.95
    train_size = X_train.shape[0]
In [15]:
# Model Define

class SignsModel(object):
    """Road Sign Classifier"""
    
    def __init__(self, config):
        self.batch_size = config.batch_size
        self.image_size = config.image_size
        self.num_classes = config.num_classes
        self.num_channels = config.num_channels
        
        self.train_size = config.train_size
        
        self.config = config
        
        self.add_placeholders()
        
        self.output_logits = self.add_model()
        
        self.loss = self.add_loss_op(self.output_logits)
        
        self.train_op = self.add_training_op(self.loss)
        
        self.prediction = self.add_prediction(self.output_logits)
    
    def add_placeholders(self):
        """Add Placeholders"""
        self.input_placeholder = tf.placeholder(tf.float32,
                                                shape=[None, self.image_size, self.image_size, self.num_channels],
                                                name="Inputs")
        self.labels_placeholder = tf.placeholder(tf.int64, shape=[None,], name="Labels")
        self.dropout_placeholder = tf.placeholder(tf.float32, name="Dropout")
    
    def add_model(self):
        """Model Definition"""

        fc1_size = self.config.fc1_size
        
        self.params = {}

        
        conv1_depth = 32
        conv2_depth = 64
        
        # Define variables
        self.params["conv1_weights"] = conv1_weights = tf.Variable(
            tf.truncated_normal([5, 5, self.num_channels, conv1_depth],
                                stddev=0.1,
                                dtype=tf.float32), name="conv1_weights")
        self.params["conv1_biases"] = conv1_biases = tf.Variable(tf.zeros([conv1_depth], dtype=tf.float32),
                                                                name="conv1_biases")
        
        self.params["conv2_weights"] = conv2_weights = tf.Variable(
            tf.truncated_normal([5, 5, conv1_depth, conv2_depth],
                                stddev=0.1,
                                dtype=tf.float32), name="conv2_weights")
        self.params["conv2_biases"] = conv2_biases = tf.Variable(tf.zeros([conv2_depth], dtype=tf.float32),
                                                                name="conv2_biases")

        
        self.params["W1"] = W1 = tf.Variable(tf.truncated_normal([self.image_size // 4 * self.image_size // 4 * conv2_depth, self.num_classes]), name="W1")
        self.params["b1"] = b1 = tf.Variable(tf.truncated_normal([self.num_classes]), "b1")
        
        # Define ops
        
        # Conv Layer 1
        conv = tf.nn.conv2d(self.input_placeholder,
                            conv1_weights,
                            strides=[1, 1, 1, 1],
                            padding='SAME')
        relu = tf.nn.relu(tf.nn.bias_add(conv, conv1_biases))
        pool = tf.nn.max_pool(relu,
                              ksize=[1, 2, 2, 1],
                              strides=[1, 2, 2, 1],
                              padding='SAME')
        
        # Conv Layer 2
        conv = tf.nn.conv2d(pool,
                            conv2_weights,
                            strides=[1, 1, 1, 1],
                            padding='SAME')
        relu = tf.nn.relu(tf.nn.bias_add(conv, conv2_biases))
        pool = tf.nn.max_pool(relu,
                              ksize=[1, 2, 2, 1],
                              strides=[1, 2, 2, 1],
                              padding='SAME')

        
        # Reshape for fully connected layers
        pshape = pool.get_shape().as_list()
        pool_line = tf.reshape(pool, [-1, pshape[1] * pshape[2] * pshape[3]])
        
        # Used only during training by setting dropout param to config.keep_prob
        pool_line = tf.nn.dropout(pool_line, self.dropout_placeholder)
        
        output = tf.matmul(pool_line, W1) + b1
      
        return output

        
    def add_loss_op(self, output):
        """Loss Op"""
        loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
                output, self.labels_placeholder))

        # Regularization
        regularizers = tf.nn.l2_loss(self.params['W1']) + tf.nn.l2_loss(self.params['b1'])
        loss += 5e-4 * regularizers

        return loss
        
    def add_training_op(self, loss):
        """Add Training Op"""
        self.global_step = tf.Variable(0, dtype=tf.float32, name="global_step")
        self.learning_rate = tf.train.exponential_decay(
            self.config.lr,
            self.global_step * self.config.batch_size,
            self.train_size, # Train size
            self.config.lr_decay,
            staircase=True
        )
        optimizer = tf.train.AdamOptimizer(self.learning_rate)
        train_op = optimizer.minimize(loss, global_step=self.global_step)
        return train_op
    
    def add_prediction(self, output):
        prediction = tf.nn.softmax(output)
        return prediction
    
    def eval_predictions(self, session, data):
        batch_size = self.config.batch_size
        num_classes = self.config.num_classes
        size = data.shape[0]
        
        if size < batch_size:
            feed_dict = {
                    self.input_placeholder: data,
                    self.dropout_placeholder: 1.0
            }
            return session.run(self.prediction, feed_dict=feed_dict)

        predictions = np.ndarray(shape=(size, num_classes), dtype=np.float32)
        
        steps_range = tnrange(0, size, batch_size, desc="Eval pred") # tqdm progress
        
        for begin in steps_range:
            end = begin + batch_size
            if end <= size:
                feed_dict = {
                    self.input_placeholder: data[begin:end],
                    self.dropout_placeholder: 1.0
                }
                predictions[begin:end, :] = session.run(self.prediction,
                                                        feed_dict=feed_dict)
            else:
                feed_dict = {
                    self.input_placeholder: data[-batch_size:],
                    self.dropout_placeholder: 1.0
                }
                batch_predictions = session.run(self.prediction,
                                               feed_dict=feed_dict)
                predictions[begin:] = batch_predictions[begin - size]
        return predictions
    
    def eval_prediction_labels(self, session, data):
        predictions = self.eval_predictions(session, data)
        return np.argmax(predictions, axis=1)
    
    def eval_prediction_labels_k(self, session, data, k=1):
        predictions = self.eval_predictions(session, data)
        values, indices = session.run(tf.nn.top_k(predictions, k=k))
        return values, indices
    
    def eval_accuracy(self, session, data, labels):
        predictions = self.eval_predictions(session, data)
        sum_ones = np.sum(np.argmax(predictions, axis=1) == labels)
        print("sum_ones =", sum_ones, " of len =", len(predictions))
        accuracy = sum_ones / len(predictions)
        return accuracy
    
    def run_epoch(self, session, data, labels, train_op = None, verbose = None):
        config = self.config
        batch_size = config.batch_size
        keep_prob = config.keep_prob
        num_steps = len(data) // batch_size
        
        if not train_op:
            train_op = tf.no_op()
            keep_prob = 1.0
            print("keep_prob =", keep_prob)
            steps_range = tnrange(num_steps, desc="Eval") # tqdm progress
        else:
            steps_range = tnrange(num_steps, desc="Steps") # tqdm progress
        
        total_loss = []
        for step in steps_range:
            x = data[step*batch_size:(step+1)*batch_size]
            y = labels[step*batch_size:(step+1)*batch_size]
            
            feed_dict = {
                self.input_placeholder: x,
                self.labels_placeholder: y,
                self.dropout_placeholder: keep_prob
            }
            _, loss, prediction = session.run([train_op, self.loss, self.prediction], feed_dict=feed_dict)
#             print("- conv =", conv)
#             print("- loss =", loss)
#             print("- logits =", logits)
#             print("- prediction =", prediction)
#             print("- prediction1 =", prediction[:, np.argmax(prediction, axis=1)])
#             print("- prediction max =", np.argmax(prediction, axis=1))
#             print("- y =", y)
            total_loss.append(loss)
#             print("loss =", loss)
            if verbose is not None and (step + 1) % verbose == 0:
                print("%d/%d loss = %f" % (step, num_steps, np.mean(total_loss)))
#             if step > 80: break
            
        return np.mean(total_loss)
    
    
    def train_model(self, session, train_data, train_labels, valid_data, valid_labels):
        max_epoch = self.config.max_epoch
        
        total_train_loss = []
        total_valid_loss = []
        epoch_valid_loss = 0
        for epoch in tnrange(max_epoch, desc="Epochs"):
            
            epoch_train_loss = self.run_epoch(session, train_data, train_labels, train_op = self.train_op, verbose = 20)
            total_train_loss.append(epoch_train_loss)
            
            epoch_valid_loss = self.run_epoch(session, valid_data, valid_labels)
            total_valid_loss.append(epoch_valid_loss)
            
            if epoch % 1 == 0 or epoch + 1 == max_epoch:
                print("Epoch %d/%d: " % (epoch + 1, max_epoch))
                print("Loss: %f" % epoch_train_loss)
                print("Validation Loss = %f" % (epoch_valid_loss))
                lr = session.run(self.learning_rate)
                print("LR =", lr)

                
        x_lab = np.arange(max_epoch)
        
        fig, ax = plt.subplots()
        ax.plot(x_lab, total_train_loss, '', label='Train Loss')
        ax.plot(x_lab, total_valid_loss, '', label='Valid Loss')
        ax.legend(loc='upper right')
        ax.set_title("Loss over Epochs")
        plt.show()
        
            
        return epoch_valid_loss
            
            
            
print("X_train_norm.shape =", X_train_norm.shape)
print("y_train_norm.shape =", y_train.shape)
print("X_valid_norm.shape =", X_valid_norm.shape)
print("y_valid_norm.shape =", y_valid.shape)
print("X_test_norm.shape =", X_test_norm.shape)
print("y_test_norm.shape =", y_test.shape)
   
# Reset graph so to have each time the same variable (useful for store/restore)
tf.reset_default_graph()
        
config = SmallConfig()
model = SignsModel(config)

# To train model from scratch change this to False
restore = False
restore_file = 'params/model_params_201611241412.ckpt'

# Create a session
session = tf.Session()
saver = tf.train.Saver(write_version=1)

# Restore or Initialize params
if restore and restore_file:
    saver.restore(session, restore_file)
    print("Model restored from ", restore_file)
else:
    # Init new
    all_variables = tf.all_variables()
    init = tf.initialize_variables(all_variables)
    session.run(init)
    
    print("Initialized new model params.")
    print("Train new model ...")
    model.train_model(session, X_train_norm, y_train, X_valid_norm, y_valid)
    
    save_time = time.strftime("%Y%m%d%H%M")
    saved_path = saver.save(session, "model_params_%s.ckpt" % save_time)
    print("Model params saved to ", saved_path)



print("Checking model test accuracy:")
accuracy = model.eval_accuracy(session, X_test_norm, y_test)
print("Test Accuracy = %.4f" % accuracy)
X_train_norm.shape = (77400, 32, 32, 3)
y_train_norm.shape = (77400,)
X_valid_norm.shape = (25800, 32, 32, 3)
y_valid_norm.shape = (25800,)
X_test_norm.shape = (25800, 32, 32, 3)
y_test_norm.shape = (25800,)
Initialized new model params.
Train new model ...
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 44.037457
39/604 loss = 38.092731
59/604 loss = 34.716202
79/604 loss = 32.112343
99/604 loss = 29.920174
119/604 loss = 28.013216
139/604 loss = 26.319201
159/604 loss = 24.801630
179/604 loss = 23.422401
199/604 loss = 22.168041
219/604 loss = 21.030027
239/604 loss = 19.995504
259/604 loss = 19.048391
279/604 loss = 18.176273
299/604 loss = 17.384058
319/604 loss = 16.653276
339/604 loss = 15.977195
359/604 loss = 15.350592
379/604 loss = 14.770714
399/604 loss = 14.233746
419/604 loss = 13.732061
439/604 loss = 13.264888
459/604 loss = 12.825408
479/604 loss = 12.412398
499/604 loss = 12.021598
519/604 loss = 11.656277
539/604 loss = 11.311864
559/604 loss = 10.986817
579/604 loss = 10.680423
599/604 loss = 10.393219
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 1/35: 
Loss: 10.337085
Validation Loss = 1.382222
LR = 0.01
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 1.873737
39/604 loss = 1.879849
59/604 loss = 1.832489
79/604 loss = 1.784428
99/604 loss = 1.746505
119/604 loss = 1.710114
139/604 loss = 1.665290
159/604 loss = 1.631704
179/604 loss = 1.606265
199/604 loss = 1.580733
219/604 loss = 1.555586
239/604 loss = 1.533013
259/604 loss = 1.509005
279/604 loss = 1.490256
299/604 loss = 1.471310
319/604 loss = 1.453623
339/604 loss = 1.437335
359/604 loss = 1.420130
379/604 loss = 1.402865
399/604 loss = 1.384884
419/604 loss = 1.370850
439/604 loss = 1.359253
459/604 loss = 1.347133
479/604 loss = 1.333041
499/604 loss = 1.317957
519/604 loss = 1.304106
539/604 loss = 1.291992
559/604 loss = 1.280646
579/604 loss = 1.270195
599/604 loss = 1.260734
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 2/35: 
Loss: 1.258900
Validation Loss = 0.624925
LR = 0.0095
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.951859
39/604 loss = 0.934549
59/604 loss = 0.914215
79/604 loss = 0.911542
99/604 loss = 0.903949
119/604 loss = 0.907973
139/604 loss = 0.898366
159/604 loss = 0.889377
179/604 loss = 0.882127
199/604 loss = 0.877062
219/604 loss = 0.875521
239/604 loss = 0.867047
259/604 loss = 0.860361
279/604 loss = 0.858611
299/604 loss = 0.855851
319/604 loss = 0.850884
339/604 loss = 0.844992
359/604 loss = 0.842494
379/604 loss = 0.842218
399/604 loss = 0.837782
419/604 loss = 0.837771
439/604 loss = 0.834918
459/604 loss = 0.832801
479/604 loss = 0.833016
499/604 loss = 0.832322
519/604 loss = 0.829827
539/604 loss = 0.827221
559/604 loss = 0.823371
579/604 loss = 0.821777
599/604 loss = 0.819556
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 3/35: 
Loss: 0.818257
Validation Loss = 0.407978
LR = 0.009025
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.723275
39/604 loss = 0.711542
59/604 loss = 0.686305
79/604 loss = 0.694836
99/604 loss = 0.693501
119/604 loss = 0.702005
139/604 loss = 0.698505
159/604 loss = 0.699008
179/604 loss = 0.697648
199/604 loss = 0.695394
219/604 loss = 0.694409
239/604 loss = 0.692428
259/604 loss = 0.690189
279/604 loss = 0.688950
299/604 loss = 0.693179
319/604 loss = 0.693726
339/604 loss = 0.692110
359/604 loss = 0.690146
379/604 loss = 0.689996
399/604 loss = 0.688564
419/604 loss = 0.686775
439/604 loss = 0.684717
459/604 loss = 0.683725
479/604 loss = 0.683504
499/604 loss = 0.681613
519/604 loss = 0.681101
539/604 loss = 0.680578
559/604 loss = 0.678444
579/604 loss = 0.676606
599/604 loss = 0.676375
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 4/35: 
Loss: 0.675964
Validation Loss = 0.345126
LR = 0.00857375
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.653625
39/604 loss = 0.633136
59/604 loss = 0.628262
79/604 loss = 0.623951
99/604 loss = 0.628732
119/604 loss = 0.634203
139/604 loss = 0.624522
159/604 loss = 0.626402
179/604 loss = 0.625859
199/604 loss = 0.623445
219/604 loss = 0.622342
239/604 loss = 0.622130
259/604 loss = 0.619116
279/604 loss = 0.617568
299/604 loss = 0.618954
319/604 loss = 0.621342
339/604 loss = 0.621808
359/604 loss = 0.620409
379/604 loss = 0.619389
399/604 loss = 0.618538
419/604 loss = 0.617838
439/604 loss = 0.618012
459/604 loss = 0.619953
479/604 loss = 0.619915
499/604 loss = 0.616261
519/604 loss = 0.617465
539/604 loss = 0.617362
559/604 loss = 0.617845
579/604 loss = 0.617137
599/604 loss = 0.617874
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 5/35: 
Loss: 0.617288
Validation Loss = 0.315967
LR = 0.00814506
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.605417
39/604 loss = 0.589624
59/604 loss = 0.568271
79/604 loss = 0.568928
99/604 loss = 0.568071
119/604 loss = 0.573546
139/604 loss = 0.565041
159/604 loss = 0.563377
179/604 loss = 0.565677
199/604 loss = 0.567334
219/604 loss = 0.567609
239/604 loss = 0.565325
259/604 loss = 0.562748
279/604 loss = 0.564363
299/604 loss = 0.565862
319/604 loss = 0.565556
339/604 loss = 0.564774
359/604 loss = 0.564675
379/604 loss = 0.564747
399/604 loss = 0.564379
419/604 loss = 0.565324
439/604 loss = 0.566109
459/604 loss = 0.569973
479/604 loss = 0.570798
499/604 loss = 0.569210
519/604 loss = 0.568979
539/604 loss = 0.571251
559/604 loss = 0.571509
579/604 loss = 0.571977
599/604 loss = 0.571612
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 6/35: 
Loss: 0.571122
Validation Loss = 0.312724
LR = 0.00773781
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.585043
39/604 loss = 0.569670
59/604 loss = 0.550905
79/604 loss = 0.558127
99/604 loss = 0.555961
119/604 loss = 0.558929
139/604 loss = 0.556853
159/604 loss = 0.560382
179/604 loss = 0.555828
199/604 loss = 0.554259
219/604 loss = 0.558052
239/604 loss = 0.553708
259/604 loss = 0.555175
279/604 loss = 0.553961
299/604 loss = 0.552789
319/604 loss = 0.552083
339/604 loss = 0.551035
359/604 loss = 0.546987
379/604 loss = 0.545633
399/604 loss = 0.544608
419/604 loss = 0.545574
439/604 loss = 0.546649
459/604 loss = 0.548329
479/604 loss = 0.547340
499/604 loss = 0.545607
519/604 loss = 0.546799
539/604 loss = 0.546815
559/604 loss = 0.544839
579/604 loss = 0.545796
599/604 loss = 0.545396
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 7/35: 
Loss: 0.544682
Validation Loss = 0.264520
LR = 0.00735092
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.530730
39/604 loss = 0.524533
59/604 loss = 0.522748
79/604 loss = 0.521974
99/604 loss = 0.518383
119/604 loss = 0.519432
139/604 loss = 0.516632
159/604 loss = 0.516311
179/604 loss = 0.513790
199/604 loss = 0.514055
219/604 loss = 0.515348
239/604 loss = 0.513880
259/604 loss = 0.511091
279/604 loss = 0.511094
299/604 loss = 0.508815
319/604 loss = 0.510904
339/604 loss = 0.511847
359/604 loss = 0.511127
379/604 loss = 0.510024
399/604 loss = 0.508391
419/604 loss = 0.507875
439/604 loss = 0.507449
459/604 loss = 0.507623
479/604 loss = 0.507722
499/604 loss = 0.506371
519/604 loss = 0.507939
539/604 loss = 0.507579
559/604 loss = 0.505867
579/604 loss = 0.505669
599/604 loss = 0.505762
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 8/35: 
Loss: 0.505247
Validation Loss = 0.255370
LR = 0.00698337
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.499311
39/604 loss = 0.484438
59/604 loss = 0.473550
79/604 loss = 0.482198
99/604 loss = 0.483001
119/604 loss = 0.485797
139/604 loss = 0.482944
159/604 loss = 0.482549
179/604 loss = 0.483479
199/604 loss = 0.482173
219/604 loss = 0.485275
239/604 loss = 0.482061
259/604 loss = 0.482905
279/604 loss = 0.484920
299/604 loss = 0.485362
319/604 loss = 0.481825
339/604 loss = 0.480076
359/604 loss = 0.478567
379/604 loss = 0.479229
399/604 loss = 0.477735
419/604 loss = 0.478493
439/604 loss = 0.477303
459/604 loss = 0.477921
479/604 loss = 0.478203
499/604 loss = 0.476943
519/604 loss = 0.478959
539/604 loss = 0.478337
559/604 loss = 0.476888
579/604 loss = 0.477539
599/604 loss = 0.479116
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 9/35: 
Loss: 0.478974
Validation Loss = 0.278243
LR = 0.0066342
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.497878
39/604 loss = 0.474343
59/604 loss = 0.473003
79/604 loss = 0.474885
99/604 loss = 0.483552
119/604 loss = 0.484202
139/604 loss = 0.475509
159/604 loss = 0.471558
179/604 loss = 0.465928
199/604 loss = 0.460109
219/604 loss = 0.458454
239/604 loss = 0.457277
259/604 loss = 0.455369
279/604 loss = 0.455930
299/604 loss = 0.455607
319/604 loss = 0.455187
339/604 loss = 0.455915
359/604 loss = 0.455922
379/604 loss = 0.454296
399/604 loss = 0.453980
419/604 loss = 0.454007
439/604 loss = 0.454589
459/604 loss = 0.455565
479/604 loss = 0.456125
499/604 loss = 0.455026
519/604 loss = 0.456933
539/604 loss = 0.458726
559/604 loss = 0.458545
579/604 loss = 0.458762
599/604 loss = 0.459683
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 10/35: 
Loss: 0.459115
Validation Loss = 0.231867
LR = 0.00630249
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.460229
39/604 loss = 0.458936
59/604 loss = 0.451094
79/604 loss = 0.457778
99/604 loss = 0.455140
119/604 loss = 0.457905
139/604 loss = 0.456117
159/604 loss = 0.453708
179/604 loss = 0.448802
199/604 loss = 0.444303
219/604 loss = 0.442829
239/604 loss = 0.440153
259/604 loss = 0.437378
279/604 loss = 0.435977
299/604 loss = 0.437956
319/604 loss = 0.435733
339/604 loss = 0.436205
359/604 loss = 0.436176
379/604 loss = 0.437294
399/604 loss = 0.438668
419/604 loss = 0.440127
439/604 loss = 0.440159
459/604 loss = 0.441770
479/604 loss = 0.443622
499/604 loss = 0.442435
519/604 loss = 0.443372
539/604 loss = 0.442073
559/604 loss = 0.440487
579/604 loss = 0.439198
599/604 loss = 0.439446
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 11/35: 
Loss: 0.439015
Validation Loss = 0.205586
LR = 0.00598737
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.384169
39/604 loss = 0.396675
59/604 loss = 0.396494
79/604 loss = 0.405813
99/604 loss = 0.416630
119/604 loss = 0.416790
139/604 loss = 0.408657
159/604 loss = 0.403378
179/604 loss = 0.401939
199/604 loss = 0.398832
219/604 loss = 0.401137
239/604 loss = 0.399163
259/604 loss = 0.399231
279/604 loss = 0.397785
299/604 loss = 0.396987
319/604 loss = 0.397382
339/604 loss = 0.395241
359/604 loss = 0.395478
379/604 loss = 0.395439
399/604 loss = 0.396264
419/604 loss = 0.398803
439/604 loss = 0.399112
459/604 loss = 0.401561
479/604 loss = 0.402377
499/604 loss = 0.402023
519/604 loss = 0.404131
539/604 loss = 0.404051
559/604 loss = 0.403156
579/604 loss = 0.402563
599/604 loss = 0.401890
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 12/35: 
Loss: 0.401676
Validation Loss = 0.202283
LR = 0.005688
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.406931
39/604 loss = 0.385215
59/604 loss = 0.375562
79/604 loss = 0.384506
99/604 loss = 0.387257
119/604 loss = 0.387684
139/604 loss = 0.390584
159/604 loss = 0.388411
179/604 loss = 0.386592
199/604 loss = 0.386713
219/604 loss = 0.389778
239/604 loss = 0.389548
259/604 loss = 0.389742
279/604 loss = 0.392270
299/604 loss = 0.392829
319/604 loss = 0.390611
339/604 loss = 0.390722
359/604 loss = 0.390018
379/604 loss = 0.388233
399/604 loss = 0.388510
419/604 loss = 0.390479
439/604 loss = 0.391405
459/604 loss = 0.392526
479/604 loss = 0.393889
499/604 loss = 0.393949
519/604 loss = 0.394278
539/604 loss = 0.393687
559/604 loss = 0.392912
579/604 loss = 0.394265
599/604 loss = 0.395294
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 13/35: 
Loss: 0.394722
Validation Loss = 0.200908
LR = 0.0054036
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.388357
39/604 loss = 0.389493
59/604 loss = 0.388617
79/604 loss = 0.389518
99/604 loss = 0.393908
119/604 loss = 0.393820
139/604 loss = 0.390103
159/604 loss = 0.383231
179/604 loss = 0.378946
199/604 loss = 0.377830
219/604 loss = 0.380169
239/604 loss = 0.377683
259/604 loss = 0.378925
279/604 loss = 0.377365
299/604 loss = 0.379770
319/604 loss = 0.379002
339/604 loss = 0.377586
359/604 loss = 0.376396
379/604 loss = 0.375245
399/604 loss = 0.374080
419/604 loss = 0.375039
439/604 loss = 0.374969
459/604 loss = 0.375301
479/604 loss = 0.374667
499/604 loss = 0.373391
519/604 loss = 0.374174
539/604 loss = 0.373223
559/604 loss = 0.372766
579/604 loss = 0.373820
599/604 loss = 0.374790
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 14/35: 
Loss: 0.374432
Validation Loss = 0.191375
LR = 0.00513342
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.342259
39/604 loss = 0.349492
59/604 loss = 0.347411
79/604 loss = 0.354573
99/604 loss = 0.356088
119/604 loss = 0.359697
139/604 loss = 0.354827
159/604 loss = 0.350002
179/604 loss = 0.353281
199/604 loss = 0.355287
219/604 loss = 0.358266
239/604 loss = 0.357158
259/604 loss = 0.358078
279/604 loss = 0.359004
299/604 loss = 0.359979
319/604 loss = 0.357766
339/604 loss = 0.356160
359/604 loss = 0.355976
379/604 loss = 0.355994
399/604 loss = 0.355126
419/604 loss = 0.354804
439/604 loss = 0.355492
459/604 loss = 0.356796
479/604 loss = 0.357011
499/604 loss = 0.355480
519/604 loss = 0.356533
539/604 loss = 0.356470
559/604 loss = 0.356842
579/604 loss = 0.357667
599/604 loss = 0.357483
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 15/35: 
Loss: 0.357077
Validation Loss = 0.174975
LR = 0.00487675
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.344552
39/604 loss = 0.341365
59/604 loss = 0.334977
79/604 loss = 0.348940
99/604 loss = 0.354172
119/604 loss = 0.354767
139/604 loss = 0.351471
159/604 loss = 0.347332
179/604 loss = 0.347170
199/604 loss = 0.343781
219/604 loss = 0.344497
239/604 loss = 0.341992
259/604 loss = 0.341785
279/604 loss = 0.339340
299/604 loss = 0.336857
319/604 loss = 0.334780
339/604 loss = 0.333380
359/604 loss = 0.332041
379/604 loss = 0.331914
399/604 loss = 0.331915
419/604 loss = 0.332563
439/604 loss = 0.334620
459/604 loss = 0.335532
479/604 loss = 0.336018
499/604 loss = 0.334630
519/604 loss = 0.336033
539/604 loss = 0.337138
559/604 loss = 0.337582
579/604 loss = 0.337816
599/604 loss = 0.338592
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 16/35: 
Loss: 0.338288
Validation Loss = 0.180341
LR = 0.00463291
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.342620
39/604 loss = 0.339521
59/604 loss = 0.342589
79/604 loss = 0.357229
99/604 loss = 0.355228
119/604 loss = 0.349208
139/604 loss = 0.345898
159/604 loss = 0.339217
179/604 loss = 0.339670
199/604 loss = 0.338544
219/604 loss = 0.340409
239/604 loss = 0.342379
259/604 loss = 0.341277
279/604 loss = 0.337805
299/604 loss = 0.337320
319/604 loss = 0.337183
339/604 loss = 0.336431
359/604 loss = 0.335736
379/604 loss = 0.334642
399/604 loss = 0.334545
419/604 loss = 0.335024
439/604 loss = 0.334326
459/604 loss = 0.334394
479/604 loss = 0.334443
499/604 loss = 0.333446
519/604 loss = 0.334273
539/604 loss = 0.333187
559/604 loss = 0.332812
579/604 loss = 0.332183
599/604 loss = 0.332747
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 17/35: 
Loss: 0.332747
Validation Loss = 0.171172
LR = 0.00440127
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.347949
39/604 loss = 0.336236
59/604 loss = 0.337308
79/604 loss = 0.338728
99/604 loss = 0.337947
119/604 loss = 0.338333
139/604 loss = 0.334889
159/604 loss = 0.333696
179/604 loss = 0.331044
199/604 loss = 0.327218
219/604 loss = 0.327315
239/604 loss = 0.323507
259/604 loss = 0.322237
279/604 loss = 0.317795
299/604 loss = 0.316189
319/604 loss = 0.315222
339/604 loss = 0.314155
359/604 loss = 0.314026
379/604 loss = 0.313030
399/604 loss = 0.311864
419/604 loss = 0.311206
439/604 loss = 0.310750
459/604 loss = 0.311183
479/604 loss = 0.310279
499/604 loss = 0.308660
519/604 loss = 0.310506
539/604 loss = 0.309963
559/604 loss = 0.310333
579/604 loss = 0.310124
599/604 loss = 0.309612
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 18/35: 
Loss: 0.309366
Validation Loss = 0.162733
LR = 0.0041812
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.276602
39/604 loss = 0.288257
59/604 loss = 0.281459
79/604 loss = 0.290993
99/604 loss = 0.299294
119/604 loss = 0.302284
139/604 loss = 0.299511
159/604 loss = 0.297359
179/604 loss = 0.296279
199/604 loss = 0.297073
219/604 loss = 0.300724
239/604 loss = 0.299866
259/604 loss = 0.300823
279/604 loss = 0.301196
299/604 loss = 0.299133
319/604 loss = 0.298970
339/604 loss = 0.299330
359/604 loss = 0.298401
379/604 loss = 0.298667
399/604 loss = 0.298687
419/604 loss = 0.299308
439/604 loss = 0.299832
459/604 loss = 0.301446
479/604 loss = 0.301184
499/604 loss = 0.300307
519/604 loss = 0.301782
539/604 loss = 0.302520
559/604 loss = 0.302478
579/604 loss = 0.302526
599/604 loss = 0.302724
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 19/35: 
Loss: 0.302884
Validation Loss = 0.154304
LR = 0.00397214
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.286528
39/604 loss = 0.278488
59/604 loss = 0.275625
79/604 loss = 0.285256
99/604 loss = 0.283584
119/604 loss = 0.290245
139/604 loss = 0.286813
159/604 loss = 0.285138
179/604 loss = 0.284762
199/604 loss = 0.284996
219/604 loss = 0.285665
239/604 loss = 0.283931
259/604 loss = 0.283546
279/604 loss = 0.281917
299/604 loss = 0.282119
319/604 loss = 0.281914
339/604 loss = 0.281610
359/604 loss = 0.281323
379/604 loss = 0.280205
399/604 loss = 0.280976
419/604 loss = 0.279673
439/604 loss = 0.281082
459/604 loss = 0.281754
479/604 loss = 0.282279
499/604 loss = 0.282627
519/604 loss = 0.283943
539/604 loss = 0.283714
559/604 loss = 0.283114
579/604 loss = 0.282972
599/604 loss = 0.283503
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 20/35: 
Loss: 0.283216
Validation Loss = 0.144891
LR = 0.00377354
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.269257
39/604 loss = 0.271966
59/604 loss = 0.280218
79/604 loss = 0.282886
99/604 loss = 0.282973
119/604 loss = 0.282658
139/604 loss = 0.280727
159/604 loss = 0.278621
179/604 loss = 0.278365
199/604 loss = 0.278253
219/604 loss = 0.277537
239/604 loss = 0.275191
259/604 loss = 0.274612
279/604 loss = 0.275366
299/604 loss = 0.277040
319/604 loss = 0.275411
339/604 loss = 0.273697
359/604 loss = 0.272401
379/604 loss = 0.270801
399/604 loss = 0.270111
419/604 loss = 0.269600
439/604 loss = 0.269798
459/604 loss = 0.270620
479/604 loss = 0.270682
499/604 loss = 0.270151
519/604 loss = 0.270002
539/604 loss = 0.269468
559/604 loss = 0.268330
579/604 loss = 0.268030
599/604 loss = 0.268042
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 21/35: 
Loss: 0.267786
Validation Loss = 0.140673
LR = 0.00358486
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.253228
39/604 loss = 0.267017
59/604 loss = 0.257793
79/604 loss = 0.264832
99/604 loss = 0.262177
119/604 loss = 0.268768
139/604 loss = 0.266897
159/604 loss = 0.265651
179/604 loss = 0.267597
199/604 loss = 0.266508
219/604 loss = 0.265298
239/604 loss = 0.263889
259/604 loss = 0.261648
279/604 loss = 0.262605
299/604 loss = 0.262581
319/604 loss = 0.261985
339/604 loss = 0.261835
359/604 loss = 0.263217
379/604 loss = 0.261751
399/604 loss = 0.262269
419/604 loss = 0.262196
439/604 loss = 0.261615
459/604 loss = 0.262073
479/604 loss = 0.261375
499/604 loss = 0.261529
519/604 loss = 0.261788
539/604 loss = 0.261668
559/604 loss = 0.261569
579/604 loss = 0.261678
599/604 loss = 0.261047
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 22/35: 
Loss: 0.260739
Validation Loss = 0.138261
LR = 0.00340562
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.259966
39/604 loss = 0.250044
59/604 loss = 0.249746
79/604 loss = 0.251570
99/604 loss = 0.255542
119/604 loss = 0.254678
139/604 loss = 0.248614
159/604 loss = 0.247383
179/604 loss = 0.246415
199/604 loss = 0.242909
219/604 loss = 0.243941
239/604 loss = 0.241914
259/604 loss = 0.240591
279/604 loss = 0.240014
299/604 loss = 0.238112
319/604 loss = 0.237431
339/604 loss = 0.237130
359/604 loss = 0.237367
379/604 loss = 0.237491
399/604 loss = 0.237236
419/604 loss = 0.238054
439/604 loss = 0.237779
459/604 loss = 0.238706
479/604 loss = 0.240056
499/604 loss = 0.239823
519/604 loss = 0.241629
539/604 loss = 0.241187
559/604 loss = 0.241553
579/604 loss = 0.241782
599/604 loss = 0.242024
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 23/35: 
Loss: 0.241669
Validation Loss = 0.125984
LR = 0.00323533
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.243517
39/604 loss = 0.243801
59/604 loss = 0.246000
79/604 loss = 0.244416
99/604 loss = 0.242684
119/604 loss = 0.241380
139/604 loss = 0.239337
159/604 loss = 0.237921
179/604 loss = 0.236830
199/604 loss = 0.233590
219/604 loss = 0.235763
239/604 loss = 0.236271
259/604 loss = 0.235919
279/604 loss = 0.234906
299/604 loss = 0.234989
319/604 loss = 0.232870
339/604 loss = 0.232047
359/604 loss = 0.231415
379/604 loss = 0.229676
399/604 loss = 0.229950
419/604 loss = 0.230303
439/604 loss = 0.230575
459/604 loss = 0.231811
479/604 loss = 0.232257
499/604 loss = 0.231480
519/604 loss = 0.232149
539/604 loss = 0.232312
559/604 loss = 0.231487
579/604 loss = 0.231359
599/604 loss = 0.232369
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 24/35: 
Loss: 0.232427
Validation Loss = 0.135991
LR = 0.00307357
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.251894
39/604 loss = 0.231547
59/604 loss = 0.226814
79/604 loss = 0.230261
99/604 loss = 0.228881
119/604 loss = 0.230389
139/604 loss = 0.228708
159/604 loss = 0.226370
179/604 loss = 0.225417
199/604 loss = 0.225586
219/604 loss = 0.227367
239/604 loss = 0.226488
259/604 loss = 0.226246
279/604 loss = 0.225040
299/604 loss = 0.224607
319/604 loss = 0.223552
339/604 loss = 0.221346
359/604 loss = 0.220876
379/604 loss = 0.219985
399/604 loss = 0.220510
419/604 loss = 0.221209
439/604 loss = 0.221075
459/604 loss = 0.222082
479/604 loss = 0.221870
499/604 loss = 0.221580
519/604 loss = 0.222481
539/604 loss = 0.222086
559/604 loss = 0.221605
579/604 loss = 0.221945
599/604 loss = 0.221935
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 25/35: 
Loss: 0.221993
Validation Loss = 0.118320
LR = 0.00291989
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.228030
39/604 loss = 0.223630
59/604 loss = 0.214739
79/604 loss = 0.218014
99/604 loss = 0.216346
119/604 loss = 0.220164
139/604 loss = 0.221652
159/604 loss = 0.218668
179/604 loss = 0.215837
199/604 loss = 0.213572
219/604 loss = 0.213592
239/604 loss = 0.212440
259/604 loss = 0.211383
279/604 loss = 0.211532
299/604 loss = 0.211423
319/604 loss = 0.211610
339/604 loss = 0.212020
359/604 loss = 0.211942
379/604 loss = 0.211606
399/604 loss = 0.211624
419/604 loss = 0.212030
439/604 loss = 0.212414
459/604 loss = 0.211790
479/604 loss = 0.211311
499/604 loss = 0.210549
519/604 loss = 0.210797
539/604 loss = 0.210906
559/604 loss = 0.210674
579/604 loss = 0.211113
599/604 loss = 0.211545
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 26/35: 
Loss: 0.211245
Validation Loss = 0.120750
LR = 0.00277389
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.218226
39/604 loss = 0.218688
59/604 loss = 0.216067
79/604 loss = 0.215667
99/604 loss = 0.215303
119/604 loss = 0.214635
139/604 loss = 0.210156
159/604 loss = 0.209367
179/604 loss = 0.206074
199/604 loss = 0.205052
219/604 loss = 0.205911
239/604 loss = 0.203825
259/604 loss = 0.202711
279/604 loss = 0.201431
299/604 loss = 0.201654
319/604 loss = 0.201999
339/604 loss = 0.201410
359/604 loss = 0.201176
379/604 loss = 0.201063
399/604 loss = 0.200886
419/604 loss = 0.201325
439/604 loss = 0.201458
459/604 loss = 0.201752
479/604 loss = 0.202011
499/604 loss = 0.201954
519/604 loss = 0.202408
539/604 loss = 0.201709
559/604 loss = 0.201496
579/604 loss = 0.202491
599/604 loss = 0.202654
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 27/35: 
Loss: 0.202542
Validation Loss = 0.113602
LR = 0.0026352
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.218374
39/604 loss = 0.214065
59/604 loss = 0.204588
79/604 loss = 0.201638
99/604 loss = 0.203039
119/604 loss = 0.204836
139/604 loss = 0.201980
159/604 loss = 0.199322
179/604 loss = 0.197568
199/604 loss = 0.197467
219/604 loss = 0.197713
239/604 loss = 0.198183
259/604 loss = 0.198002
279/604 loss = 0.197114
299/604 loss = 0.195435
319/604 loss = 0.194871
339/604 loss = 0.195179
359/604 loss = 0.195105
379/604 loss = 0.194228
399/604 loss = 0.194457
419/604 loss = 0.194606
439/604 loss = 0.194863
459/604 loss = 0.196225
479/604 loss = 0.196380
499/604 loss = 0.195180
519/604 loss = 0.195850
539/604 loss = 0.196216
559/604 loss = 0.196082
579/604 loss = 0.196662
599/604 loss = 0.196535
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 28/35: 
Loss: 0.196426
Validation Loss = 0.102799
LR = 0.00250344
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.173593
39/604 loss = 0.179671
59/604 loss = 0.182028
79/604 loss = 0.179654
99/604 loss = 0.183684
119/604 loss = 0.184860
139/604 loss = 0.185646
159/604 loss = 0.184717
179/604 loss = 0.183332
199/604 loss = 0.183213
219/604 loss = 0.183095
239/604 loss = 0.182947
259/604 loss = 0.184976
279/604 loss = 0.185154
299/604 loss = 0.186862
319/604 loss = 0.185229
339/604 loss = 0.185096
359/604 loss = 0.185159
379/604 loss = 0.184963
399/604 loss = 0.184576
419/604 loss = 0.185017
439/604 loss = 0.185903
459/604 loss = 0.185991
479/604 loss = 0.186257
499/604 loss = 0.185965
519/604 loss = 0.187290
539/604 loss = 0.186825
559/604 loss = 0.186837
579/604 loss = 0.187375
599/604 loss = 0.188203
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 29/35: 
Loss: 0.188231
Validation Loss = 0.100830
LR = 0.00237827
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.204873
39/604 loss = 0.194704
59/604 loss = 0.193560
79/604 loss = 0.194831
99/604 loss = 0.197915
119/604 loss = 0.201685
139/604 loss = 0.199187
159/604 loss = 0.198455
179/604 loss = 0.196160
199/604 loss = 0.195059
219/604 loss = 0.193520
239/604 loss = 0.191722
259/604 loss = 0.190299
279/604 loss = 0.190134
299/604 loss = 0.189214
319/604 loss = 0.187618
339/604 loss = 0.186691
359/604 loss = 0.186435
379/604 loss = 0.186147
399/604 loss = 0.185856
419/604 loss = 0.186187
439/604 loss = 0.186365
459/604 loss = 0.186300
479/604 loss = 0.186055
499/604 loss = 0.185822
519/604 loss = 0.186091
539/604 loss = 0.185438
559/604 loss = 0.185061
579/604 loss = 0.185241
599/604 loss = 0.185663
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 30/35: 
Loss: 0.185649
Validation Loss = 0.096928
LR = 0.00225935
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.192352
39/604 loss = 0.189882
59/604 loss = 0.182287
79/604 loss = 0.184422
99/604 loss = 0.183268
119/604 loss = 0.183294
139/604 loss = 0.183726
159/604 loss = 0.181822
179/604 loss = 0.181074
199/604 loss = 0.179066
219/604 loss = 0.179925
239/604 loss = 0.180049
259/604 loss = 0.179434
279/604 loss = 0.178904
299/604 loss = 0.179917
319/604 loss = 0.179705
339/604 loss = 0.179484
359/604 loss = 0.178941
379/604 loss = 0.178794
399/604 loss = 0.177643
419/604 loss = 0.178157
439/604 loss = 0.178029
459/604 loss = 0.178149
479/604 loss = 0.177982
499/604 loss = 0.177789
519/604 loss = 0.177776
539/604 loss = 0.177824
559/604 loss = 0.177146
579/604 loss = 0.178006
599/604 loss = 0.178145
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 31/35: 
Loss: 0.178024
Validation Loss = 0.098519
LR = 0.00214639
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.159112
39/604 loss = 0.173750
59/604 loss = 0.173173
79/604 loss = 0.176864
99/604 loss = 0.179231
119/604 loss = 0.181059
139/604 loss = 0.177049
159/604 loss = 0.176434
179/604 loss = 0.176765
199/604 loss = 0.175871
219/604 loss = 0.176952
239/604 loss = 0.176173
259/604 loss = 0.174566
279/604 loss = 0.174368
299/604 loss = 0.173848
319/604 loss = 0.173669
339/604 loss = 0.172155
359/604 loss = 0.173086
379/604 loss = 0.172839
399/604 loss = 0.171842
419/604 loss = 0.171355
439/604 loss = 0.171603
459/604 loss = 0.171069
479/604 loss = 0.170708
499/604 loss = 0.170924
519/604 loss = 0.171622
539/604 loss = 0.171863
559/604 loss = 0.172201
579/604 loss = 0.172259
599/604 loss = 0.172278
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 32/35: 
Loss: 0.172120
Validation Loss = 0.091248
LR = 0.00203907
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.176384
39/604 loss = 0.167967
59/604 loss = 0.171142
79/604 loss = 0.170488
99/604 loss = 0.167654
119/604 loss = 0.168185
139/604 loss = 0.168226
159/604 loss = 0.166607
179/604 loss = 0.164141
199/604 loss = 0.163329
219/604 loss = 0.163580
239/604 loss = 0.163149
259/604 loss = 0.162750
279/604 loss = 0.162740
299/604 loss = 0.162018
319/604 loss = 0.162246
339/604 loss = 0.160662
359/604 loss = 0.160384
379/604 loss = 0.160747
399/604 loss = 0.160849
419/604 loss = 0.161092
439/604 loss = 0.161159
459/604 loss = 0.161626
479/604 loss = 0.161736
499/604 loss = 0.161605
519/604 loss = 0.161687
539/604 loss = 0.162374
559/604 loss = 0.161914
579/604 loss = 0.162386
599/604 loss = 0.162954
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 33/35: 
Loss: 0.162899
Validation Loss = 0.090094
LR = 0.00193711
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.171196
39/604 loss = 0.170117
59/604 loss = 0.170948
79/604 loss = 0.170703
99/604 loss = 0.166951
119/604 loss = 0.166616
139/604 loss = 0.163963
159/604 loss = 0.161247
179/604 loss = 0.159270
199/604 loss = 0.159551
219/604 loss = 0.161018
239/604 loss = 0.159739
259/604 loss = 0.158512
279/604 loss = 0.158193
299/604 loss = 0.157496
319/604 loss = 0.157508
339/604 loss = 0.156709
359/604 loss = 0.157502
379/604 loss = 0.157404
399/604 loss = 0.157772
419/604 loss = 0.158015
439/604 loss = 0.157860
459/604 loss = 0.158292
479/604 loss = 0.158501
499/604 loss = 0.157228
519/604 loss = 0.157635
539/604 loss = 0.157512
559/604 loss = 0.157412
579/604 loss = 0.157494
599/604 loss = 0.157367
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 34/35: 
Loss: 0.157124
Validation Loss = 0.085777
LR = 0.00184026
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
19/604 loss = 0.144531
39/604 loss = 0.146392
59/604 loss = 0.146891
79/604 loss = 0.150418
99/604 loss = 0.157284
119/604 loss = 0.157615
139/604 loss = 0.154945
159/604 loss = 0.153250
179/604 loss = 0.152227
199/604 loss = 0.150996
219/604 loss = 0.152625
239/604 loss = 0.152512
259/604 loss = 0.153172
279/604 loss = 0.153011
299/604 loss = 0.153737
319/604 loss = 0.153834
339/604 loss = 0.153095
359/604 loss = 0.152529
379/604 loss = 0.152688
399/604 loss = 0.152185
419/604 loss = 0.152036
439/604 loss = 0.152207
459/604 loss = 0.152865
479/604 loss = 0.153345
499/604 loss = 0.152967
519/604 loss = 0.153355
539/604 loss = 0.153594
559/604 loss = 0.153189
579/604 loss = 0.153059
599/604 loss = 0.153081
keep_prob = 1.0
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
Epoch 35/35: 
Loss: 0.153068
Validation Loss = 0.085127
LR = 0.00174825

Model params saved to  model_params_201611250258.ckpt
Checking model test accuracy:
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
sum_ones = 25555  of len = 25800
Test Accuracy = 0.9905
In [16]:
# Test Accuracies
print("Checking test accuracy:")
accuracy = model.eval_accuracy(session, X_test_norm, y_test)
print("Test Accuracy = %.4f" % accuracy)

print("Checking valid accuracy:")
accuracy = model.eval_accuracy(session, X_valid_norm, y_valid)
print("Valid Accuracy = %.4f" % accuracy)

print("Checking train accuracy:")
accuracy = model.eval_accuracy(session, X_train_norm, y_train)
print("Train Accuracy = %.4f" % accuracy)
Checking test accuracy:
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
sum_ones = 25555  of len = 25800
Test Accuracy = 0.9905
Checking valid accuracy:
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
sum_ones = 25573  of len = 25800
Valid Accuracy = 0.9912
Checking train accuracy:
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
sum_ones = 77154  of len = 77400
Train Accuracy = 0.9968

Question 3

What does your final architecture look like? (Type of model, layers, sizes, connectivity, etc.) For reference on how to build a deep neural network using TensorFlow, see Deep Neural Network in TensorFlow from the classroom.

Answer:

In [17]:
### Train your model here.
### Feel free to use as many code cells as needed.
In [18]:
# Confusion Matrix Evaluation

def confusion_matrix(labels, pred, num_classes):
    cm = np.zeros([num_classes, num_classes])
    for l,p in zip(labels, pred):
        cm[l,p] += 1
    s = np.sum(cm, axis=1)
    r = cm / s.reshape([len(s),1])
    return r
    
# Borrowed from http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py
def plot_confusion_matrix(cm, classes,
                          normalize=True,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    """    

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, "%.2f" % cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

    
    
prediction_labels = model.eval_prediction_labels(session, X_test_norm)
print("prediction_labels =", prediction_labels)

cm = confusion_matrix(y_test, prediction_labels, model.config.num_classes)
    
plt.figure(figsize=(22,22))
plot_confusion_matrix(cm, classes=range(model.config.num_classes),
                      title='Confusion matrix')
Widget Javascript not detected.  It may not be installed properly. Did you enable the widgetsnbextension? If not, then run "jupyter nbextension enable --py --sys-prefix widgetsnbextension"
prediction_labels = [ 0 24 12 ..., 28 28 28]
In [19]:
# plt.figure(figsize=(22,22))
print("len =", len(y_test))
print("matches =", np.sum(prediction_labels == y_test))
# plot_confusion_matrix(cm, classes=range(model.config.num_classes),
#                       title='Confusion matrix')
len = 25800
matches = 25555
In [20]:
# print(prediction_labels[:20])
# print(y_test[:20])

Question 4

How did you train your model? (Type of optimizer, batch size, epochs, hyperparameters, etc.)

Answer:

Question 5

What approach did you take in coming up with a solution to this problem?

Answer:


Step 3: Test a Model on New Images

Take several pictures of traffic signs that you find on the web or around you (at least five), and run them through your classifier on your computer to produce example results. The classifier might not recognize some local signs but it could prove interesting nonetheless.

You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.

Implementation

Use the code cell (or multiple code cells, if necessary) to implement the first step of your project. Once you have completed your implementation and are satisfied with the results, be sure to thoroughly answer the questions that follow.

In [21]:
### Load the images and plot them here.
### Feel free to use as many code cells as needed.

def resize_image(img):
    im = cv2.resize(img, (32,32))
    im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)
    return im


def load_images_from_folder(folder):
    images = []
    for filename in os.listdir(folder):
        img = cv2.imread(os.path.join(folder,filename))
        if img is not None:
            images.append(resize_image(img))
    return np.array(images)

# Load images from folder
imgs = load_images_from_folder('test_imgs')

imgs_norm = normalize(imgs)

vals, imgs_pred_labels = model.eval_prediction_labels_k(session, imgs_norm, k=5)
# print("imgs_pred_labels =", imgs_pred_labels)
# print("vals =", vals)


num = 0
for idx, val, im in zip(imgs_pred_labels, vals, imgs):
    num += 1
    plt.figure(figsize=(2,2))
    plt.imshow(im)
    plt.title(num)
    text = []
    for i,v in zip(idx, val):
        text.append("%.5f - %s\n" % (v, signnames[i]))
    plt.text(35, 32, "".join(text))

Question 6

Choose five candidate images of traffic signs and provide them in the report. Are there any particular qualities of the image(s) that might make classification difficult? It would be helpful to plot the images in the notebook.

Answer:

In [22]:
### Run the predictions here.
### Feel free to use as many code cells as needed.

Question 7

Is your model able to perform equally well on captured pictures or a live camera stream when compared to testing on the dataset?

Answer:

In [23]:
### Visualize the softmax probabilities here.
### Feel free to use as many code cells as needed.

Question 8

Use the model's softmax probabilities to visualize the certainty of its predictions, tf.nn.top_k could prove helpful here. Which predictions is the model certain of? Uncertain? If the model was incorrect in its initial prediction, does the correct prediction appear in the top k? (k should be 5 at most)

Answer:

Question 9

If necessary, provide documentation for how an interface was built for your model to load and classify newly-acquired images.

Answer:

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to \n", "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

In [ ]: